MemGator - A Portable Concurrent Memento Aggregator

With the growth in the number of public web archives it is becoming important to provide a means to aggregate them for better coverage and completeness. The Memento protocol [2] provides a uniform API to lookup URIs in web archives. Due to the wide support of the Memento protocol in the archiving ecosystem it is now easy to aggregate their holdings for any URI lookup. However, current applications can either use their custom aggregator implementation or rely on centralized services such as LANL’s Time Travel portal and ODU Memento Aggregator. While centralized third party services are serving their purpose well, the convenience has the trade-off of lack of customization and control such as the client application cannot choose which archives to be aggregated. Centralized services are usually good for general usages, but not suitable for specialized purposes such as research or heavy traffic applications. For example, certain archives have IP-based traffic throttling policies which might limit the ability of the centralized server in case of heavy traffic. Similarly, the recent surge of OldWeb.today caused increased load on archives, as a result one archive requested its exclusion from being polled. This would have been an issue if they were using a centralized service. There are a few open source aggregator implementations such as Memento Server and Memento Java Client Library, but they are either outdated or require a server setup. With these issues in mind, we created MemGator that provides a standalone cross-platform binary without any external dependencies. It can be used as a one-off command to retrieve the response on the standard output or run as a web service to replicate necessary features of the centralized Memento aggregator services. We tried to keep the service API as close as possible to the LANL’s Time Travel service for greater interoperability. Both the modes (CLI and server) come with a handful of customization options that are documented in the binary itself and can be seen using standard help flag. One such configuration option is to supply a custom list of archives to be aggregated or use the archive profile based archive ranking to query top-K archives only. The tool is currently being used heavily in OldWeb.today, WAIL, Mink, and our internal archiving research projects. It has proved to be reliable even in the extreme load conditions. An aggregator is a good example of a concurrent application. It relies on various upstream archives which consumes the maximum amount of the overall time in network I/O while the process sits idle. Performing this operation sequentially will make it useless as the number of upstream